The Switching Method: Elaborations

The switching method [4] is a scheme which combines two universal source coding algorithms. The two universal source coding algorithms both estimate the probability distribution of the source symbols, and the switching method allows an encoder to choose which of the two probability distributions it uses for every source symbol. The switching algorithm is an efficient weighting algorithm that uses this switching method. This paper focuses on the companion algorithm, the algorithm running in parallel to the main CTW-algorithm. 1 The switching method: A short introduction The switching method [4] defines a way in which two modeling algorithms can be combined. Consider a source sequence x1, . . . , xN . Suppose that two sequential modeling algorithms, A and B, run both along the entire source sequence, and give for every symbol an estimate of its probability distribution. These modeling algorithms could be memoryless estimators, estimators for fixed tree models, or entire universal source coding algorithms on their own. At each moment the encoder in the switching method uses the estimate from only one of the two modeling algorithms to encode the next source symbol. The estimate of the other modeling algorithm is ignored, but the statistics of both modeling algorithms are updated. The switching method starts using modeling algorithm A. It can switch from one modeling algorithm to the other one between any two source symbols. The switching behaviour of the switching method is specified for the decoder with a transition sequence t1, . . . , tN . If a transition symbol ti = 1 then the encoder switched from one modeling algorithm to the other one between source symbols xi−1 and xi. Otherwise ti = 0. The transition sequence will be intertwined with the source sequence, t1, x1, . . . , tN , xN , to allow sequential encoding and decoding. This combined sequence will then be encoded. In principle any encoding algorithm can be used to encode the transition sequence. Once an encoding scheme is chosen, the best transition sequence can be found. Note that because both the source sequence and the transition sequence have to be encoded, the best transition sequence should minimize the cost of describing both. Instead of trying to find the best transition sequence, it is possible to weight over all transition sequences in a relatively efficient way, by choosing the encoding of the transition sequence in a specific way. For example, using fixed probabilities for a switch (ti = 1) and a non-switch results in a scheme with a computational complexity only linear in the length of the source sequence. But because the optimal value for the probability of a switch can only be found after scanning the entire source sequence, this would result in a two-pass algorithm. The switching algorithm [4] uses a binary memoryless estimator, the Krichevsky-Trofimov estimator, to estimate the probabilities of the transition sequence. It has a computational complexity quadratical in the length of the source sequence. The switching algorithm uses the switching method at the highest level: the two modeling algorithms are universal source coding algorithms. If the two universal source coding algorithms are sufficiently different, then the local performance of these two algorithm can differ significantly, and the switching algorithm can obtain a considerable gain in performance by being able to switch between them locally. But if the two universal source coding algorithms are very similar, switching between them during the encoding process may not result in a gain in performance. So the switching algorithm should use different source coding algorithms. It can use a wide variety of algorithms, like the context-tree weighting algorithm (CTW-algorithm), PPMD, PPM∗, or LempelZiv algorithms once they are redesigned to estimate probability distributions. In this paper we will use the CTW algorithm as the main modeling algorithm, algorithm A. The first part of this paper will focus on a possible companion algorithm: an algorithm which behaves like a cross between PPM∗ and Lempel-Ziv’77. 2 The companion algorithm 2.1 Some considerations The switching method achieves its best performance if the two universal source coding algorithms are “complementary”. We applied both PPMD and PPM∗ as companion algorithm, the algorithm running in parallel with CTW. PPM∗ results in far superior performance. The reason for this is the internal workings of these algorithms. PPMD resembles the CTW-algorithm in the way in which they model the source sequence. PPM∗ on the other hand has a more Lempel-Ziv’77 behaviour. The advantage of a Lempel-Ziv’77 type of algorithm (see [1] for an overview of Lempel-Ziv algorithms) is that it can encode a long substring with one (short) codeword the second time that this substring occurs in the source sequence. In the CTW-algorithm the first occurrence of this substring results in a few single counts at the deeper nodes of the context tree. But before the CTW-algorithm recognizes these deeper nodes as a part of the model, these nodes have to be visited many times. Thus when the substring occurs for the second time the deeper nodes do not influence the weighted probability significantly yet, and the second occurrence of the substring will be encoded less efficiently than by a Lempel-Ziv’77 type of algorithm. Considering the already excellent performance of PPM∗ as companion algorithm to CTW, it is interesting to develop a PPM∗ algorithm that behaves even more as a Lempel-Ziv’77 algorithm. 2.2 PPM∗ and Lempel-Ziv’77 PPM∗ [2] is a PPM-algorithm with an unbounded context length. Before encoding a new symbol, it first selects the context length (the order). It chooses the shortest deterministic context (a context each time followed by the same symbol so far), or otherwise the longest context. From this selected context length on it performs the normal PPM-escape mechanism. For every context s from the selected length down to the empty context, the PPM-escape mechanism splits the (remaining) code space in two parts. A fraction 1−PESC of the code space is used to encode the symbols following this context s. Once this is done, PPM∗ escapes to the one symbol shorter suffix of the context and applies the same process to the remaining fraction of the code space, PESC . After encoding the symbols following the empty context (thus the memoryless estimates), PPM∗ escapes to the last model, the order -1 model. This model assigns a uniform probability to all symbols, and it is used to encode symbols not seen so far. The escape probability PESC is recomputed for every context length and for every new symbol. It is a function of several variables, like the number of symbols following that context, the number of different symbols, etc. The purpose of the escape probability is (more or less) to reserve a fraction of the code space for symbols not following the current context. s x y active past phrases